3. Results¶
3.1. Search Grid With 5-fold Cross Validation¶
The model was fine tuned by finding optimal values for the following hyperparmeters:1) L2 regularization and 2) dropout rate applied to each of the hidden layers, and 3) learning_rate.
Optimal hyperparameters were found by doing a full grid-search over 75 different combination of the three hyperparameteres and cross-validating each with a 5-fold cross-validation method. A total of 75 models were trained and validated. Following plot shows the performance of each model in terms of its training and validation accuracies. The models are shown in a descending order their mean validation accuracy. The error bars indicate standard deviation across folds.
Thirty-ninth model was the best performing model with mean training and validation accuracy of 0.89 and 0.82, respectively, and was chosen as the final model to be tested on the held out test set. Final model hyperparameters were: 1) L2 = 0.003, dropout = 0.3, and learning_rate = 0.001
3.2. Test Accuracy¶
The trained model was tested on near-miss segments of the 19 held-out participants. Following figure shows temporal and overall accuracy on the held-out participants. The model performs resonably well from the 1st timepoint (TP) itself, with a mean accuracy of 0.8. The mean accuracy steadily increases to 0.89 at the 7th TP. “Overall” accuracy is the mean accuracy across TP, which is 0.83.
3.3. Chance Accuracy¶
A classifier with the best hyperparameters was trained on the training set with randomly shuffled labels k_perm number of times. After every training iteration, the classifier was tested on a validation set with “non-shuffled” (i.e., true) labels. This process was meant to simulate a chance accuracy distribution. The mean of the chance accuracies was used as a baseline performance measure against the observed performance of the classifer when it was trained on the training set with true labels.
Test accuracy: Observed > Null (p = 0.0099)
